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ABSTRACT 

The objective of this study was to validate a scale for assessment of academic projects. 
As a complement, we examined its predictive ability by comparing the scores of 
advised/corrected projects based on the model and the final scores awarded to the work 
by an examining panel (approximately 10 months after the project design). 

Results of Confirmatory Factor Analysis (with 161 projects) indicated convergent validity 
and discriminant validity. 

Regression analysis revealed a positive and significant relationship between the scores of 
projects and final scores of the work, strengthening the claim of the usefulness of the 
model to minimize potential wear in advising processes. 

Keywords: Project evaluation; scale; advising processes. 

INTRODUCTION 

Academic Papers: 

A Disagreement about the Advising Process 

Advising an academic work consists of a monitoring process whose main objective is to 
assure that the study carried out by the advisee may have a level of quality consistent 
with other studies produced and published in their field of research, so as to allow 
comparability and identify their contribution. There are indications that the advisor has a 
significant role in this process (see, eg Wygal & Slout, 1989; Fernandes, Klering & Aguiar, 
1993; Martins, 1997; Freitas, 1998; Freitas, 2002; Leite Filho & Martins, 2006, Ferreira, 
Furtado & Silveira, 2009; Dias, Patrus & Magalhaes, 2011; Mizany, Khabiri & Sajadi, 
2012). Mentoring, voluntarily or forcibly (Martins, 1997), became part of the range of 
activities developed by teachers. This fact is present especially in stricto sensu post¬ 
graduate studies, but this situation has been no different when looking at the scenario of 
undergraduate courses. 
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Thus, there are arguments that not all advisors are or have been adequately prepared to 
develop an orientation process (Martins, 1997; Flecha, 2003; Leite Filho & Martins, 2006; 
Dias, Patrus & Magalhaes, 2011). "Methodology manuals teach what a monograph should 
be, but do not teach how the teacher should guide the student so he or she may have the 
internal coherence and articulation required for scientific work" (Dias, Patrus & 
Magalhaes, 2011, p.715). 

Specifically in graduation Course Conclusion Papers (CCP), some students, because they 
like to read and write or have already accumulated experience with the development of 
articles in Scientific Initiation projects, for example, may require less from their advisors 
regarding their orientation skills than those students in the opposite extreme. 
"Depending on the level of the student's autonomy, that is, his or her experience in 
research, the role of the advisor will be more or less intense, diverse and frequent 
(Ferreira, Furtado & Silveira, 2009, p.171). But this does not indicate that, by itself, the 
interest and ease of reading and writing of student will ensure a quality academic work, 
because if the advisor does not have this ability developed, the CCP can be seriously 
compromised, not being good enough for approval in an examining panel. 

It is noteworthy to point one of the basic assumptions that is usually made about 
advisors: the academic advisor, having gone through a formal orientation process (in 
which he or she was receiving orientation) and elaboration of a CCP (be it at the Masters 
or Doctorate level), "would be able to teach research skills" (Leite Filho & Martins, 2006, 
p.101). However, as Dias, Patrus & Magalhaes (2011, p.698) point out: "It is not 
uncommon, among peers, for a masters or doctorate student to refer to his or her advisor 
as a 'disorientator'. The suffering of the learner in the process of preparing his or her 
monograph, dissertation or thesis is witnessed by all those present in academia", and this 
lack of satisfaction also mentioned by Mizany, Khabiri and Sajadi (2012). The fact that 
the teacher was once an advisee does not necessarily make him or her an apt to advice. 

An Alternative to Minimize Tension In The Advising Process 

In response to the contextualized clash, Dias, Patrus and Magalhaes (2011, p.701) 
offered "an alternative orientation of academic papers, through clear parameters on the 
characteristics that the work to be performed by the student must have". It is a 
spreadsheet containing items that mainly constitute projects of the Administration area. 
In the model presented by the authors, these items are grouped in the following major 
topics: 


> for research projects: "Introduction"; "Theoretical Foundation" and 
"Methodology". 

> for the final work: besides the three topics mentioned above, "Analysis and 
Discussion of Results" and "Conclusions and Suggestions" are included. 

In their study, Dias, Patrus and Magalhaes (2011) explore in detail each of the topics and 
their sub-topics, permitting the application of the model more objectively and also 
allowing potential doubts about its content to be clarified. 


Thus, there is the understanding that, with the application of this model in the analysis of 
projects and CCPs (whether monographs, dissertations or theses), the absence of 
parameters in the advising process is minimized, which may contribute to increased 
advice quality and, consequently, the quality (and quantity) of the advised and defended 
papers. 

Dias, Patrus and Magalhaes (2011, p700) show that the model is part of sharing "an 
experience that is succeeding in the process of academic advising of master's and 
doctoral level papers". Even with this indication of model fitness, it is understood that a 
formal analysis of its convergent validity, discriminant validity, internal consistency and 
reliability, would make the proposal even more robust for application in other studies and 
practical research activities. The authors stated that: "when submitting an article of this 
nature, it is hoped that colleagues from the academic community will contribute 
suggestions, and critical experiments. Other models are possible" (Dias, Patrus & 
Magalhaes, 2011, p. 715). 

Purpose of the Study And Choice Of Research Institution 

The analysis of convergent validity, discriminant validity, reliability and internal 
consistency of a model is possible by means of statistics applied to it, examples of them 
being: Exploratory Factor Analysis, Confirmatory Factor Analysis, Cronbach's alpha, the 
Composite Reliability Index, among other indicators. 

With the quantification of each of the items of the model proposed by Dias, Patrus and 
Magalhaes (2011), it is possible to apply this statistic, which is within the scope of this 
study, and is its overall purpose: to quantitatively validate a scale for the evaluation of 
academic projects. Considering that the authors' proposed model can be used to evaluate 
full papers or projects, we also established an additional objective for this study, which 
is: to analyze the predictive ability of the model by comparing the resulting grades of the 
advised projects based on the model and the final grades given to the projects by a panel 
composed of different teachers (which are generally three for graduation and master's 
and five for doctoral projects). 

The implementation of this research, based on the goals set, was possible because of the 
identification of a federal institution of higher education having in its Accounting course 
curriculum (semester system), the disciplines of CCP-01, CCP-02 and CCP-03 (Course 
Conclusion Paper 1, 2 and 3, respectively). It is the School of Accounting Sciences of the 
Federal University of Uberlandia (FACIC/UFU), which is located in the city of Uberlandia 
(State of Minas Gerais, Brazil). 

This college, that completed 50 years in 2013 and recorded over 2000 graduates in the 
Accounting course, has 07 administrative staff and 29 effective teachers. In the next 3 
years, at least 50% of its faculty will have a doctoral degree (currently, this percentage is 
31.03%). In 2013 the institution began offering the Academic Master of Science in 
Accounting, with 16 regular and 4 special students in this first class. 
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In this institution, one teacher per class gives the subject of CCP-01. The purpose of this 
course is for the student to have, in its conclusion, a research project which will be 
formally accompanied by a teacher in CCP-02 (for review of the theoretical framework, 
methodology, data collection and initial analysis of the results) and CCP-03 (for 
finalization of results analysis, closing statements, reviewing the work and presentation 
to a panel composed of at least three teachers). Thus, we identified the possibility of 
applying the model in the CCP-01 course (to examine the convergent validity, 
discriminant validity, reliability and internal consistency, in statistical terms) and relate 
its results to grades awarded by a panel approximately 10 months after the initial 
analysis and design guidelines (CCP-03). This application is possible because in the 
subject of CCP-01, the monitoring of the teacher, for advising and evaluating students, is 
done by means of an instrument whose topics correspond to the model of Dias, Patrus 
and Magalhaes (2011). 

In order to facilitate the analysis of what was developed in this study, we drew Figure: 1 
below. It also implies a positive relationship between the grades on the evaluation of the 
project and the grades awarded by the panel in the presentation of the final papers. 
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Figure: 1 

Diagram illustrating the analysis developed in this study 

Notes: CCP-01: Subject of Course Conclusion Paper 01, semestral, taught by one teacher per class, in which the 
final requirement of the course is the presentation of a research project to continue in CCP-02 and CCP-03. 
CCP-02: Subject of Course Conclusion Paper 02, semestral, where each student is accompanied by his or her 
respective advisors, and at the end of the semester a grade is assigned by the advisor. CCP-03: Subject of 
Course Conclusion Paper 03, semestral, where the student should, with the help of the advisor, complete his or 
her CCP and present it before a panel of at least three teachers. These subjects are taught at FACIC/UFU. 


To achieve the general objective, the following specific objectives were outlined: 

> Identify an institution that allows the implementation and monitoring of 
the model analyzed in an undergraduate, masters or doctoral degree class. 
As noted above, the institution selected for this study was the FACIC/UFU, 
undergraduate course; 
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> application of the model to the advising of research projects ; 

> evaluate the research projects based on the model, assigning grades to 
each of its items. As stated previously, the teacher responsible for the 
course of CCP-01 had evaluated (quantitatively) the projects presented in 
the course with an instrument corresponding to the one of Dias, Patrus and 
Magalhaes (2011). This database created by the teacher was fully available 
for analysis in this study; 

> examine the convergent validity, discriminant validity, reliability and 
internal consistency of the component topics of the model; 

> verify the relationship between the grades assigned to projects in CCP-01 
to the grades assigned to the final work by an examining panel in CCP-03. 

Topic Relevance and Justification 

"It is urgent to think about the advising and didactic of academia" (Dias, Patrus & 
Magalhaes, 2011, p. 715) just like "writing" is important in the educational process 
(Wygal & Stout, 1989). These statements help understand the importance of further 
studies on the subject of this work. 

A reinforcement can be found in Fernandes, Klering and Aguiar (1993, p. 103), where the 
authors state that "the difficulties in preparing dissertations and resulting student 
dropouts as well as the disappointments on the part of mentors deserve an effort to at 
least begin a discussion of enabling strategies aimed at increasing the rate of production 
of dissertations", which may also be applicable for course completion papers in general. 

According to Freitas (2002, p.93), "when a student does not complete the work, he or she 
loses what he or she could and would achieve during the credit period... For the 
institutions involved, everything will be capitalized as losses in their statistics, for the 
advisor, it will be a personal failure". 

We can add to what was already addressed the fact that the area of Teaching and 
Research in Management and Accounting has received increasing attention from 
academia in recent years. 

The demands in the preparation of academic papers have also grown significantly. A fact 
that underlies this statement is the amount of unselected papers in the subject area of 
Education and Research in Accounting at the 13 th USP Congress of Controllership and 
Accounting-2013 (19 abstracts were approved to continue the review process, but only 
one paper was selected for presentation and discussion at the congress). 

Consultation made at the portal of the Ministry of Education (MEC, 2013), in Brazil, there 
are 1,294 Undergraduate courses in Accounting and 2,686 Undergraduate Administration 
courses in activity. Given the growth of these courses in recent years, an auxiliary 
instrument to analyze and monitor the development of CCPs may be an important tool for 
the teachers to justify their analyses. 
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Moreover, a potential stress (Dias, Patrus & Magalhaes, 2011, p. 715) in an advising 
process can be minimized with the use of a more objective tool to assess/monitor the 
work, which will optimize the time of the advisor and the advisee. 

Thus, it is expected that the results of this study will contribute to the academic 
community, especially because of the project assessment tool that will be discussed, 
presented and validated. 

We also expected to contribute to future research to be conducted in this field, since, 
besides the validation proposal, an analysis is being made of a possible determinant of 
the quality of the final papers presented by students, papers which represent potential 
articles to be submitted the conferences and journals and subsequently contribute (in 
one form or another) to the construction of theory (Whetten, 1989) on the subject. 

THEORETICAL FRAMEWORK 

The process of developing an academic paper(monographs, dissertations or theses) 
requires an interaction between the advising teacher and advisee student in all stages, 
from research, analysis and discussion of the results until its final composition (Ferreira, 
Furtado & Silveira, 2009). In the academic environment, Wygal and Stout (1989, p. 246) 
claim that "writing is a complex process." An important factor in the process of writing a 
post graduate academic paper is associated with the advisor and the quality of guidance 
offered (Mizany, Khabiri & Sajadi, 2012). According to Freitas (2002, p.39), the advisor 
can be seen as a "regulatory and institutional controller, has his or her name attached to 
the work and is responsible for what the student does and especially what the student 
does not do". Therefore, it is expected that the advisor be prepared for the orientation 
process and deal with the challenges and limitations of the process, which does not 
always occur (Leite Filho & Martins, 2006; Dias, Patrus & Magalhaes, 2011). 

Ideally, the guidance should be given to qualified teachers with appropriate knowledge 
and experience and relevant scientific production (Ferreira, Furtado & Silveira, 2009). At 
this point, Pinheiro, Melkers and Youties (2012) point out that the publication of the 
work is a key indicator of research and knowledge production and the co-authoring of the 
advisor makes all the difference. But what is often perceived is a lack of preparation of 
both the advisors and the advisees (Fernandes, Klering & Aguiar, 1993). 

Writing an academic paper is not an easy task. This is a complex process that is 
influenced by several variables, such as the profile and experience of the student and the 
experience of the advisor (Wigal & Stout, 1989). 

Another factor that can influence the outcome of the work both positively or negatively is 
the very relationship between the advisor and the advisee. There are many papers 
dealing with this issue, highlighting it as one of the greatest challenges of the advising 
process (see, for example: Leite Filho & Martins, 2006; Viana, 2008; Ferreira, Furtado & 
Silveira, 2009; Barth-Teixeira et al., 2011). 
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According to Mizany, Khabiri and Sajadi (2012), the advisor is responsible for helping the 
student in choosing the theme, motivating him or her during the writing of the paper and 
establishing a good relationship with the student. Thus, the lack of harmony between the 
advisor and the advisee both in the guidance and on assessment can harm the progress 
of the work (Freitas & Zawislak, 1996). 

A study by Rodrigues Jr., Fleith and Alves (1993) on the interactions between advisor and 
advisee based on critical incidents, pointed out that in cases where the orientation 
process was unsuccessful, the advisors attribute the failure to the advisees. 

These, in turn associate the failure to the inability of the advisor to manage the process 
and the lack of knowledge about the content in addition to non-availability of the advisor. 
On the subject, Freitas and Zawislak (1996, p.7) argue that one of the factors that 
negatively affects the orientation process is the "lack of a clear definition of the 
evaluation criteria, where subjectivity reigns, rather than immediate and objective 
feedback". 

Anyhow, there are many studies that show barriers and challenges related to the 
advising process as well as the need for better-prepared advisors. Flecha (2003), for 
example, indicates the importance of providing tools for advisors to perform their role 
well. 

However, there are few works such as the one of Dias, Patrus and Magalhaes (2011) 
aimed at proposing ways to address these barriers and challenges and improve the 
orientation process. The study conducted by Dias, Patrus and Magalhaes (2011), 
proposed a model that aims to assist advisors in guiding/monitoring the development of 
academic papers. 

By aiming at extending the research originally developed and published by Dias, Patrus 
and Magalhaes (2011) and covering the lack of studies in the area, this study aims to 
validate a model for the evaluation of academic projects, and to assess its predictive 
ability. 

METHODOLOGICAL ASPECTS 
Data 

The accomplishment of the objectives of this work was made possible by contact with an 
institution of higher education, FACIC/UFU, which features a curriculum structure that 
allows the completion of the work. 

Moreover, there is also the fact that, in developing the subject that involves preparing a 
project (CCP-01), the teacher uses an instrument corresponding to the model presented 
by Dias, Patrus and Magalhaes (2011). Frame 1, below, shows a comparison between the 
instrument used by the CCP-01 teacher and model of the mentioned authors. 
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General 

Topics 

Model proposed by 

Dias, Patrus e Magalhaes (2011) 

(for projects) 

Model used in CCP-01 (for projects) 


Contextualization 

Contextualization 


Research Question 

Research Problem 

Introduction 

General purpose 

General purpose 

Specific Objectives 

Specific Objectives 


Practical justifications 

Justifications and Relevance of Topic 


Theoretical justifications 

- 

Theoretical 

Literature Review 

Theoretical - Definitions and concepts 

grounding 

Guiding issues and/or Hypotheses 

Theoretical - Previous Studies 


Research Design 

Research classification - Approach 


Population and Sample 

Research classification - Procedures 


Variable Instrumentation 

- 

Methodology 

Data Collection - Instrument 

Method of Data Collection 


Data Collection - Process 

- 


Methods of Analysis 

Method of Data Analysis 


Boundaries and Limitations 

Limitations of the Study 

- 

References 

References - According to ABNT 

- 

Appendices and Annexes 

Timetable 

- 

- 

General Structure of the Project 

- 

- 

Citations in General - According to ABNT 


Frame: 1 

Comparison of assessment instruments addressed in this study 


Source: first and second columns: adapted from Dias, Patrus and Magalhaes (2011), third column: adapted from 
a worksheet courtesy of a FACIC/UFU CCP-01 teacher. 

As Frame: 1 shows, the content of the assessment instruments are similar, since they 
assess the research project based on three main areas: Introduction, Theoretical 
Framework and Methodology, besides formal aspects. The big difference between the 
models lies in the fact that the instrument used by CCP-01 teacher considers a scale of 0 
to 10 for each item evaluated. Ie, if the Context of the Study fails to comply with its 
purpose, it scores zero. If it caters adequately, is well written and the ideas are well 
concatenated, it scores 10. If there is concatenation of ideas, but this contextualization 
does not take to the research question, the score is average, and so on, varying according 
to each case (the scores can also be fractioned, for example, 9.5). 
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According to the teacher of the course, this spreadsheet is shown to the students in two 
stages: first, after the presentation of initial discussions about the project, students 
deliver the first version to the teacher, whom, using the worksheet, assigns a score to 
each of the items for each student (this version is worth the equivalent of 5% of the 
semester grade). Based on the spreadsheet scores, students have a parameter to correct 
their project and consult the teacher. After about two weeks, students deliver the final 
version, which is again evaluated based on the spreadsheet and the score is released to 
students by registration number, which is the second stage of the disclosure of the 
spreadsheet. The score of this second version corresponds to 40% of the semester course 
grade, which is calculated proportionally based on the arithmetic average of the scores of 
its items. 


Data Collection Was Performed In the Following Sources 
CCP-01 Project Grades 

Project grades resulting from the application of the assessment instrument used by the 
teacher as well as grades concerning each of the items of the assessment instrument 
were provided by the subject teacher for the period 5 semesters (this grade corresponds 
to the final version of the project). The period was not longer because this new 
curriculum framework, involving the preparation of a CCP by students, came into effect 
at FACIC/UFU in 2007. Consequently, the first classes of the new curriculum began to 
attend CCP in 2010, since this course is offered in the 8th Period (CCP-01, 8th period, 
CCP-02, 9th Period; CCP-03, 10th Period). It is worth mentioning that CCP-01 is a 
prerequisite for CCP-02, and CCP-02 is a prerequisite for CCP-03. Besides it, as a 
prerequisite to take the CCP-01 course, students must have attended and have been 
approved in all subjects up to the 6th period. Number of valid observations: 161 
(2010/02, 2011/01, 2011/02, 2012/01, and 2012/02). 


CCP-03 Grades 

These were obtained from the Coordination of Undergraduate Accounting Sciences Office 
FACIC/UFU. In a public presentation, an examining panel, composed of at least three 
teachers, as already explained, awards this grade. Number of valid observations: 79 
(2011/02, 2012/01, and 2012/02). That is, until 2013/01, only three classes had 
attended CCP-03 and at the time of the beginning of data collection, information relating 
to 2013/01 was not available. Quantitative analysis was divided into two major steps, 
which are explained below: 

> validation of the questionnaire, and 

> analysis of the relationship between the grades of the CCP-01 projects 
with the CCP-03 grades. 


Validation of Questionnaire and Regression Analysis 

In the quantitative evaluation stage of the questionnaire, we initially applied the 
Exploratory Factor Analysis (EFA), with the objective of determining whether the 
grouping of variables would be similar to those proposed by Dias, Patrus and Magalhaes 

(2011) - 98 


In other words, the objective of this analysis was to evaluate if the components of the 
Introduction would be grouped in the same factor, if the components of the methodology 
would be grouped in another factor, and so on. 

The method used was Principal Component Analysis with Varimax rotation, accompanied 
by KMO (Kaiser-Meyer-Olkin) and Bartlett Sphericity tests, showing the fitness of data to 
the AFE. As a criterion for the number of factors to be extracted, we used the "Latent 
Root Criterion", extracting factors with eigenvalues greater than 1 (Hair et al., 2005, 

p.101). 

On the sample size. Hair et al. (2005, p-98) point out that "as a general rule, the 
minimum is to have at least five times more observations than the number of variables to 
be analyzed, and the more acceptable size would have a ratio of ten to one. Some 
researchers even propose a minimum of 20 cases for each variable". 

The instrument being evaluated contains 16 questions, which indicates a need for at least 
80 respondents and maximum of 160 (16 x 5 =80 and 16 x 10 =160). With this, the 
number of observations from CCP-01 (161, as explained previously) is in the upper limit 
of this ratio. 

As we already have a theoretical model that underlies the organization of the evaluation 
instrument, more robust procedures for analysis of convergent validity and discriminant 
validity of the instrument were applied. As explained by Kline (2011, p. 71-72), 

A set of variables presumed to measure the same construct shows 
convergent validity if their intercorreiations are at least moderate in 
magnitude. In contrast, a set of variables presumed to measure 
different constructs shows discriminant validity if their 
intercorreiations are not too high. 

Thus, with regard to convergent validity, Cronbach's Alpha and the Reliability Composite 
Index were calculated for each construct. For these two indicators, we expected to 
obtain values of at least 0.70 (Hair et al., 2005). 

However, Hair et al. (2005) also emphasize that in exploratory research, a minimum of 
0.60 for these two indicators is accepted. For the discriminant analysis, noting the study 
of Paiva, Roth and Fensterseifer (2008) as well as the recommendations of Kline (2011), 
we checked the chi-square statistic differences between pairs of factors (Introduction x 
Theoretical Framework, for example) perfectly correlated and the factors evaluated with 
free correlation. 

This procedure was repeated for the three possible combinations of factors: Introduction 
x Theoretical Framework; Introduction x Methodology; Theoretical Framework x 
Methodology. 
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The objective of this procedure is to, for example, examine whether the items used to 
assess the Introduction of the project are measuring aspects related to the introduction 
or if aspects of methodology are also present. 

Greater consistency would be found if grades for the items of methodology were part of 
the methodology group, not mixed with Introduction and Theoretical Framework. There 
may be a positive correlation between these factors, but they are not necessarily part of 
the same factor. The perfect correlation between different factors indicates specification 
problems. 

This step is part of the procedures for the Confirmatory Factor Analysis (CFA), which is 
"particularly useful in the validation of scales for the measurement of specific constructs" 
(Hair et al., 2005, p.493). 

For the CFA, the following indicators were observed, based on considerations of Byrne 
(2001) and Hair et al. (2005): 


> Chi-square statistic, which is expected to be not significant for the 
differences in anticipated and actual matrices. However, this statistical 
analysis must be done in conjunction with the others, as it is sensitive to 
sample size. 

> Comparative indexes: normed fix index (NFI), comparative fit index (CFI), 
incremental fit index (IFI) and Tucker-Lewis index (TLI), where values 
above 0.90 indicate optimal fitness. 

> Absolute measure adjustment: Indicators goodness-of-fit index (GFI). 
Recommended values are close to 0.90. 

> Root mean square error of approximation (RMSEA) statistic, with a 
recommended value below 5% in well-fit models, or below 8% for 
cautious analyses. 


Finally, to analyze the relationship between the study variables, the linear regression 
analysis by the method of Ordinary Least Squares (Hair et al., 2005, p.493) was applied, 
and their standardized residuals were analyzed for normality through the Kolmogorov- 
Smirnov test. 

RESULTS 

Questionnaire Validation 

The first step consisted of the descriptive analysis of the issues that make up the 
research model. The results are displayed in Table: 1 here. Table: 1 show that, on 
average, students receive good reviews in the final version of theirCCP-01 project 
because the average score for most items evaluated is above seven, and many of them, 
near or exceeding eight. Even 
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Table: 1 

Descriptive statistic of variables in the evaluation instrument 


Factor 

Variable/Model item 

n 

Avg. 

Std. Dev. 

Min. 

Max. 


Contextualization 

161 

7.976 

2.359 

0.000 

10.000 


Research Problem 

161 

8.665 

2.691 

0.000 

10.000 

Introd. 

General purpose 

161 

8.932 

2.056 

0.000 

10.000 


Specific Objectives 

161 

7.932 

3.224 

0.000 

10.000 


Justifications and Relevance of Topic 

161 

7.982 

2.554 

0.000 

10.000 

Theoretical 

Theoretical - Definitions and concepts 

161 

7.802 

2.434 

0.000 

10.000 

grounding 

Theoretical - Previous Studies 

161 

6.640 

3.562 

0.000 

10.000 


Research classification - Approach 

161 

8.087 

2.604 

0.000 

10.000 


Research classification - Procedures 

161 

7.783 

2.92 

0.000 

10.000 

Method. 

Method of Data Collection 

161 

7.087 

3.314 

0.000 

10.000 


Method of Data Analysis 

161 

5.796 

3.777 

0.000 

10.000 


Limitations of the Study 

161 

6.280 

4.067 

0.000 

10.000 


References - According to ABNT 

161 

7.249 

3.279 

0.000 

10.000 


Timetable 

161 

9.143 

2.159 

0.000 

10.000 

Form 

General Structure of the Project 

161 

8.056 

2.368 

0.000 

10.000 


Citations in General - According to ABh 

161 

6.957 

3.331 

0.000 

10.000 


Notes: each item of the model was evaluated based on a score of 0 to 10. 


The Factor 'Form' was included in this spreadsheet as a way to identify the last four items. 

so, there are students who scored zero in some of the items, as well as students who 
scored a 10. With this database, we proceeded to the EFA and the results are reported in 
Table 2, for the rotated matrix. 
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There was a lot of consistency in the loads displayed by the factors, with the 
correspondence available in Frame: 2 being suggested in this paper. 


Table: 2 

Results for Exploratory Factor Analysis (EFA) 


Factor 

Variable/Model item 

Factor 01 

Factor 02 

Factor 03 


Contextualization 

0.598 

0.494 

... 


Research Problem 

— 

0.715 

... 

Introd. 

General purpose 

— 

0.764 

... 


Specific Objectives 

— 

0.665 

... 


Justifications and Relevance of Topic 

... 

0.575 

... 

Theoretical 

Theoretical - Definitions and concepts 

0.492 

... 

0.531 

grounding 

Theoretical - Previous Studies 

... 

... 

0.636 


Research classification - Approach 

0.834 

... 

... 


Research classification - Procedures 

0.842 

... 

... 

Method. 

Method of Data Collection 

0.839 

... 

... 


Method of Data Analysis 

0.664 

... 

... 


Limitations of the Study 

... 

0.623 

... 


References - According to ABNT 

... 

... 

0.627 


Timetable 

_ 

_ 

0.793 

Form 

General Structure of the Project 

0.56 

— 

0.502 


Citations in General - According to ABNT 

0.518 

... 

0.495 


Notes: " refers to loads lower than 0.45, which were excluded for best viewing of the results. 


The results presented in Table: 2 suggest the creation of three factors (remembering that 
the criterion used to determine the number of factors was the "Latent Root Criterion", in 
which the factors with eigenvalues greater than lwere extracted). 


Factors 

Correspondence 

Factor 01 

Aspects related to the methodology of the study, its overall structure and 
obedience to the rules of ABNT (formatting). 

Factor 02 

Aspects of the introduction of the study and its limitations. 

Factor 03 

Aspects related to Theoretical Framework, References and timetable 


Frame: 2 

Identification of factors created by EFA 
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The three factors identified shown themselves compliant to the proposal of Dias, Patrus 
and Magalhaes (2011), which already shows an exploratory result regarding the 
instrument evaluated. However, since these results are based on the EFA, there is only 
slight evidence that this compliance is adequately measured. To measure this compliance 
with a more robust statistic was applied to the CFA. 

For the application of CFA, it was necessary to establish the test model and link each item 
to its respective factor. Figure 2 describes the bindings performed, based on the original 
model being tested and also based on the results of EFA for allocation of variables in the 
factor "Form". 


Part A Part B 
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Figure: 2 

Model used for the Confirmatory Factor Analysis (CFA) 

Notes: Indicators for the Model relative to Part B of the Figure: Chi-Squared Statistics: 178.639 with 98 degrees 
of freedom. Chi-Square weighted by the degrees of freedom: 1,823. NFI: 0.892. CFI: 0.947. IIF: 0,948. TLI: 
0.935. GFI: 0.880. RMSEA: 7.17%. RMSEA LO 90: 5.48%. HI RMSEA 90: 8.82%. 

Part A of Figure: 2 refer to the first version of the model. After analyzing the adjustment 
indicators of Part A, observed in conjunction with the modification indexes, three 
correlations were added between the error terms of the Research Classification: 

> Method of Approach with Procedure Method 

> Method of Collection with form of data analysis, and, 

> Method of Collecting data with General Structure of Paper 




































































































































The new model 

With the inclusion of these correlations, model fitting (available in Part B of Figure 2) was 
satisfactory, being: 


> Chi-Squared Statistics: despite being significant at 5% level, its value 
divided by the degrees of freedom (178.639/98=1.823) was less than 3 
(Kline, 2011), indicating a reasonable fit, which should be assessed in 
conjunction with other indicators. 

> Comparative indexes indicated optimal fittings, obtaining the following: 
NFI (0.892), CFI (0.947), IFI (0.948) and TLI (0.935). 

> Absolute fitness measure: GF1 (0.880) was close to 0.90, as indicated by 
literature. 

> The (RMSEA) statistic was 7.17%, and its confidence interval was as 
follows: 5.48% (90 LO) and 8.82% (HI 90). Thus, its lower limit was close 
to the minimum recommended, and its average value was below 8%. 


Then we calculated Cronbach's Alpha and the Reliability Composite Index, to verify the 
reliability and internal consistency of each factor of the model analyzed. 

The results are summarized in Table: 3 and indicate appropriate fitness, since the 
literature recommends levels of at least 0.70 (all measurements presented were above 
the minimum recommended). 


Table: 3 

Cronbach's Alpha and the Composite Reliability Index of the model analyzed 


Factors 

No. of 

items 

Cronbach's 

Alfa 

Composite Reliability 
Index 

Introduction 

5 

0.798 

0.976 

Theoretical Grounding 

5 

0.779 

0.962 

Methodology 

6 

0.871 

0.981 


As can be seen in Part B of Figure: 2, was inserted a correlation between the three factors 
of evaluation of projects. 

To check the level of specification of the model, ie, to verify whether the factors are 
measuring different aspects of the projects of each student, the discriminant validity was 
performed and results are available in Table: 3. 


Available statistics refer to reviews by pairs of factors. 
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Table: 3 

Results for the discriminant validity 


Pairs of Factos 

No Restrictions 

With Perfect 

Correlation 

Chi-Square Difference 

Chi- 

Square 

Chi - df 
Square 

Difference Significance 


Introduction 


Theoretical Groundir 

54.383 

34 

75.854 

35 

21.471 

<0.01 

Methodology 

83.121 

40 

135.376 

41 

52.255 

<0.01 

Theoretical Grounding 

Metholodogy 

92.751 

40 

131.468 

41 

38.717 

<0.01 


Note: read the value of 21.471 as follows: the pair of factors Introduction and Theoretical Framework 
with free correlation between them, showed a chi-square statistic of 54.383 with 34 degrees of freedom. 
This same pair of factors with the correlation between them fixed at one presented a chi-square statistic 
of 75.854 with 35 degrees of freedom. The difference between the chi-square statistics was 21.471 
(75.854 to 54.383) that with one degree of freedom is statistically significant at 1%. 


Although we included correlations between the factors "Introduction", "Methodology" 
and "Theoretical Framework" in the model in Figure 2, the discriminant analysis indicated 
that they measure different aspects within the assessment tool. Ie, they are not confused 
when assessed together and have behaviors with similar variations in assessments of 
projects of different students. 

These results are favorable to the fitness of the instrument for measuring aspects related 
to different factors of the research projects. 

Analysis of the Relationship Between 

The Grades of Projects and the Final Grades of the Work 

Having validated the assessment tool used by the CCP-01 teacher, the following 
procedure consisted in the linear regression analysis between the grades of projects and 
the final work grades in CCP-03. 

For the final grade of the project, an arithmetic average obtained from 16 evaluation 
items was considered. Table 4 shows the descriptive statistics of the two variables. 
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Table: 4 

Descriptive statistics for the grades of projects and final grades of papers 


Variables 

n 

Avg. 

Std. Dev. 

Min. 

Max 

Project Grade 

79 

7.165 

2.198 

0.000 

10.000 

CCP-03 Grade 

79 

85.468 

9.670 

40.000 

98.000 


Note: The project grade varies on a scale of 0 to 10; 
the CCP-03 grade varies on a scale of 0 to 100. 


As Table: 4 shows, there were students who scored zero on the project, as there were 
students who scored ten. As for the final grade of the work, the minimum grade awarded 
was 40 points and the maximum 98 points. Ie, there is still no record of a score of 100 by 
a panel of examiners in the analysis of CCPs, with an average score equal to 85.468. The 
result of the linear regression analysis is available in Table: 5. 

Table: 5 

Results for the linear regression analysis 


Variables 

Beta 

Std. Error 

T Statistic 

Sig. 

Constant 

69.218 

4.126 

16.774 

0.000 

Project Grade 

2.175 

0.536 

4.058 

0.000 

N 

79 

- 

- 

- 

R2 adjusted 

0.165 

- 

- 

- 

Sig.(Residuals) 

0.211 

- 

- 

- 

Sig.(F-Test) 

0.000 

- 

- 

- 


Notes: Sig. (Residuals): significance for the Kolmogorov-Smirnov test 
applied to the standard residues in the linear regression analysis. Sig. (F-Test): 
significance for the F test of the regression model. 


The results of the regression analysis assist in the test of the positive sign shown in 
Figure 1, in the introduction of this study. I.e., the beta on the grade of the project had a 
positive (2.175) and significant value at 1% level. This indicates that, on average, an 
increase of one grade unit in the project provides an increase of 2.175 CCP-03 grade 
units. 

The observed relationship between the grade of the project and the final grade of the 
work indicate that there is a positive and significant relation to the assessment 
conducted based on the instrument used by the teacher and the grade given by a panel 
composed of three different teachers. The test of normality of residuals indicates good 
fits for the regression analysis performed. 
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The results also underlie the reasoning that, on average, students with lower grades on 
projects also have a tendency to achieve lower grade on CCP- 03 (almost 10 months after 
the project evaluation). 

The main finding to be highlighted is the following: the instrument used in CCP-01 to 
advise/monitor research projects, which is consistent with that proposed by Dias, Patrus 
and Magalhaes (2011), has reliability, internal consistency, enables the measurement of 
different aspects of research projects, and the evaluation as a result of its application has 
significant positive relationship with the grades of the final works later presented before 
an examining panel. 

FINAL CONSIDERATIONS 

The advising process, although essential to the smooth running of academic activities, 
does not always culminate with the expected results for the parties involved (advisor and 
advisee). 

In order to reduce the consequences of the lack of standards in the guidance processes, 
Dias, Patrus and Magalhaes (2011) proposed a model that helps to advise/monitor the 
development of academic papers. An equivalent instrument to that proposed by these 
authors was already being used to evaluate projects in the discipline of CCP-01 at 
FACIC/UFU, this instrument being subjected to quantitative treatments in this study. 

The results showed that the instrument reported adequate levels of reliability and 
internal consistency (in statistical terms), and allowed the measurement of different 
aspects contained in research projects, which was confirmed by the analysis of 
convergent and discriminant validity of the model. 

Moreover, the results showed strong evidence that the project grades resulting from the 
application of the assessment tool had a positive and statistically significant relationship 
with the grades obtained in CCP-03 of the final works defended by the students. 

That is, on average, students who received good reviews in the project also received 
good reviews from the examiners at the time of their public presentation. 

That conclusion was possible because, in CCP-03, the final work was assessed based on 
the average grade of the examining panel, ranging from 0 to 100. If, inCCP-03, the 
criterion had been: approved or failed, without a grade representing a scalar measure, 
the analyses developed between the grades of the project and the final evaluation of the 
work would not be possible. 

This is a limiting factor for the application of this relationship (project grades with 
dissertation/thesis presentation grades) in Stricto-Sensu post-graduation, but this does 
not preclude the use of a validated instrument to advise/monitor the development of 
research projects in these courses. 
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The results allow us to extend the research originally developed and published by Dias, 
Patrus and Magalhaes (2011), also giving it more robustness. Its relevance is obvious to 
minimize potential stress in the advising processes, especially in undergraduate courses, 
environment in which the results were tested. 

The quality of guidance and the work produced by the students are the first steps to 
developments in advanced studies and publications that may allow high theoretical and 
methodological rigor which is essential for building theory (Whetten, 1989) in any area of 
knowledge. Three limiting factors are linked to the results of this study: 

> the number of observations available for analysis, because only the 
projects of five different classes were considered in the study, 

> the 161 projects were evaluated by only one teacher. 

However, the result obtained, revealing significant positive relationship between the 
grades given by the teacher and the grades awarded by different panels, indicates that 
the assessment is in line with the perceptions of different evaluators, and the study was 
conducted with a database from only one institution. 

Thus, further studies are recommended and new applications of the model in other 
higher education institutions. For future research, we suggest a (quantitative) 
application of the model in Stricto-Sensu graduate course projects, observing whether its 
reliability, internal consistency, convergent and discriminant validity will be maintained, 
as observed in this study. 

We also suggest the inclusion of additional explanatory variables in the linear regression 
analysis performed in order to increase the variability explained by the equation. 
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